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Haptics Transmission Systems 

The present invention relates to haptics transmission systems and more 
particularly to a system Incorporating improved latency correction and a method 
5 of improving latency correction. 

The present invention relates to haptic communications and more particularly to 
improving the response of haptic devices coupled by way of a 
telecommunications network. 

Tactile output from computers has been used to enhance game playing to 

10 provide a ''feer, for example vibration, thus adding an additional sensory 
perception to the, game§._§;uch outputs have also been used to enable visually 
impaired people to read documents and to feel drawings and the like. The basic 
operation of haptics output devices has been described in our published co- 
pending PCT Patent Application published as W003/007136 which disclosed a 

15 method for adapting haptic interface output characteristics to correct for per- 
person differences in the sense of touch. In a further PCT patent application, 
publication no WO03/02885 there is disclosed a method of enabling reading of 
the Moon alphabet by use of a haptics output device. In the transmission of 
character sets from computers or data stores to haptics output devices there is 

20 unlikely to be any time critical activity dependant upon the output signals. 
However, where game play is involved, particularly if .players are competing 
against each other or against the machine in a competitive manner, transmission 
delays of forward or reverse force parameters may have a significant impact on 
the sensed experience. 

25 As game play is more likely to be carried out over a connectionless 

network, for example the internet or world wide web, rather than by a point to 
point communications link, signal latency may be introduced which can result in 
an inconsistency in the sensed movement of the output compared with the input! 

Furthermore, sensory devices require frequent updates in both signal 

30 directions if the feel of the sensed movement and. reaction to users response is to 
be realistic. The number of updates required to maintain realism, while not a 
problem where the haptic output device is in close proximity and direct 
connection to the generating processor, may result in the communications 



wo 2005/041009 



PCT/GB2004/004025 



2 

network overloading in a very short time and/or may require extreme allocations 
of bandwidth. 

According to one aspect of the present invention there is provided a 
method of activating a haptic output device of the kind responsive to signals 
5 defining directional force comprising receiving a series of signals defining a 
multiplicity of data packets, each packet defining a position measured at one 
location for transmission to the current location, determining from packet data the 
information defining a position to which a haptic output device is expected to 
move, storing historic positional data defining each of a multiplicity of positions to 
10 which the haptic output device has moved, deriving a data model of the space in 
which directional forces are being applied at said one location and storing data 
defining said model, deriving from the historic positional data and the data 
defining the model an anticipated position and generating output signals defining 
force and direction to move the haptic output device towards said anticipated 
1 5 position and correcting for differences^ between the anticipated position and the 
transmitted position on receipt of subsequent positional data. 

Preferably the method includes signalling In each direction whereby haptic 
forces applied at one device in reaction to an applied force towards the current 
defined position are reflected to a corresponding device in the form of current 
20 positional signals in a series of return data packets. 

The method may include determining from the data model of the space 
the present of an impeding object whereby modification of the anticipated 
position and/or force may occur. 

A feature of the present invention provides an interactive haptic output 
25 terminal in combination with a bi-directional transmission arrangement, the 
terminal comprising at least a haptic output device and control means, said 
control means receiving signals from said haptic output device to determine a 
cun-ent position for said device, and to determine from signals received from said 
transmission arrangement a preferred current position for said haptic output 
30 device, said control mean determining an output force and direction required to 
move said haptic output device from the current position to the preferred position, 
storing historic positional data defining each of a multiplicity of positions to which 
the haptic output device has moved, deriving a model of the space in which 
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directional forces are being applied and storing data defining said model, deriving 
from the historic positional data and the data defining the model an anticipated 
position and generating output signals defining force and direction to move the 
haptic output device towards said anticipated position and correcting for 
5 differences between the anticipated position and the transmitted position on 
receipt of subsequent positibnal data. 

The control means will receive signals; from the haptic output device 
containing data defining the position of said device at any particular time and will 
convert said data to signals for transmission to said bl-directional transmission 
1 0 arrangement at predetermined intervals. 

The signals defining a prefenred cun-ent position may be generated by an 
environment simulator, for example a programmed computer, or may be generated 
by a corresponding interactive output terminal at the opposed end of the 
transmission arrangement. 
15 Where a series of packets defining prefen-ed position are received, each 

packet defining a directional force applied at one location for transmission to the 
current location, the control means may include means to determine from packet 
data the sequence of transmission and re-ordering the data Into a numerically 
correct series, extrapolating from previously received packets an anticipated linear 
20 movement to be defined by subsequently received packets and applying output 
directional force signals corresponding to said anticipated linear movement in 
respect of any missing data packet. 

A haptics transmission system in accordance with the invention will now 
be described by way of example only with reference to the accompanying 
25 drawings of which: - 

Figure 1 is a block schematic diagram of a first haptics 
communications system in which a network interconnects an environmental 
simulation to a haptics input/output device; 

Figure 2 is a block schematic diagram of a haptics communications 
30 system having a plurality of interconnected haptics input/output devices; 

Figure 3 is a schematic diagram of data interchange within the system of 
Figure 2; 
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Figure 4 is a schematic flow chart of the method of measuring latency 
between two locations to effect adjustment of the system of Figure 2; 

Figure 5 is a schematic flow chart of the method of calculating forces to 
be applied locally; 

5 Figure 6 to 8 are schematic flow charts showing how to put the invention 

in to practice. 

Referring to Figure 1, in our co-pendirig European patent application 
number 01305947.2, there is disclosed a method of providing a haptics output 
representation of a scene stored, for example, as object model data. In this case a 

10 processor 1 includes a program responsive to the position of a haptics output 
device (for example the Phantom 1.0 Haptic Output device from Sense Able 
Technologies Inc of the USA), to output reaction forces based upon the object 
model data. The object model data stored in a data store 3 could define textures, 
surfaces and locations of fixed or moveable objects which could be perceived by a 

1 5 user of the haptics output interface 2. In some further developments disclosed in 
the preceding application information held in a data store 4 based upon a player 
identity 6 allowed player preferences 7 and a gamma correction factor 8 to be 
used to provide appropriate output adjustment to ensure that different players 
have approximately the same perception of the output at the haptics output 

20 interface 2. 

As disclosed the processor 1 was closely coupled to the haptics output 
interface 2 and could therefore provide substantially continuous detection of 
location of the user's finger with respect to the x, y, z axes of the device thus 
allowing real time simulation of the environment defined by the object model data 
25 3. 

As hereinbefore mentioned, once a network 5 is introduced between the 
haptics output interface 2 and the processor 1 continuous communication of the 
virtual environment and responsive signalling determining the user's response and 
location by way of an input/output interface 6 to the processor 1 becomes 
30 impractical if one requires to update the signalling at substantially continuous 
rates. Furthermore, latency introduced to the signalling by the network results in 
an extremely jerky feel to the Information being transmitted. 
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Turning now to Figure 2, where a plurality of haptic output devices 21, 
22 are communicating by way of respective input/output interfaces 25, 26 to 
respective processors 23, 24 the problems of network latency and signalling 
limitations become more acute. Thus, if the processor 23 receives by way of a 
5 network adaptor 27 signals indicating a position for the haptic output device 22 
and instantly seeks to move the haptic device 21 to that position accordingly a 
substantial jerk in the movement will be apparent. In any event, the user of the 
haptic output device 21 will be applying a backward force which may inhibit such 
movement and therefore prevent the processor 23 from aligning the position of 

10 the haptic output device 21 with that of the haptic output device 22. 
Correspondingly, the processor 23 in measuring the location of the haptic output 
device 21 will send signals back through the network 5 by way of network 
adaptor 28 to the processor 24, which will attempt to make a corresponding 
movement in the haptic output device 22. Thus, because the communication 

15 between haptic devices 21 and 22 is no longer of a continuous mode but is 
receiving and transmitting positional information at intervals the experience of the 
users will be significantly impaired. In addition, the period of time taken for 
signals to traverse the network (network latency) will further impair user 
perception. 

20 Thus considering also Figure 3, it is possible to perceiye that if at a 

location A a user locates the haptic output device 21 "position" and receh/es a 
force from the haptic output device then the local position data derived from 
"position" is derived by the personal computer (PC) 23 and transmitted to the 
network. At the same time remote position data received from the network is 

25 translated by the PC. 23 into local force data. 

Corresponding position and force derivative data will also be used at 
location B by the PC 24. 

In a practical network the position data and force output are transmitted 
between each end at approximately five millisecond intervals. Thus each time a 
30 new position is received a force is output in an attempt to move the output device 
to a new position^ effectively with a motor pushing against the local user. In 
effect the user's at each end at positions A and B are coupled together and the 
two-way activity and data transmission effectively attempts to move both output 



wo 2005/041009 



PCT/GB2p04/004025 



devices 21 and 22 to corresponding positions. This simulates connection of the 
two users in a manner such as if they were connected together by some Icind of 
resilience device, for example/ k spring or flexible rod. Because there is a 
reactionary force at each end there can be instability between the two devices 
5 because of the feed bacic loop with deferred transmission of forces may result in 
an effective positive feedback. 

Network latency also results in a tendency for the user to feel a jerkiness 
in the response of the effector because of the delay in receiving packets by way 
of the network, particularly If variation in the latency of the network is occurring. 
1 0 This may detract from the quality of the user's experience. 

Thus, as hereinbefore described, referring again to figure 3, a system 
comprising two haptic output devices 21, 22 each attached to a personal 
computer 23,24 which are in turn linked through, say, the internet B is 
susceptible to the network latency problems outlined above. 
15 In use, each computer 23,24 reads the respective position of the haptic 

output device 21,22 attached thereto and transmits data defining the positional 
co-ordinates of the handpiece of its haptic display to the other computer which 
calculates the force required to coerce its respective handpiece of the connected 
haptic display towards the same co-ordinates. The computer therefore instructs 
20 the haptic display to exert that force on the user through the handpiece. The 
symmetric communication keeps the two displays moving in unison enabling 
transmission of simple forces, positions, shapes textures and motions to be 
transmitted between them. 

Turning now to figure 4, from each of the locations A and B the 
25 respective local clock 30 of the PCs 23,24 is used to determine the network 
latency. Thus, from location A the time from the local clock is bundled Into a 
transmission packet, step 31, and transmitted at step 32 through the network 5. 
The packet is received at location B, step 33, and is immediately retransmitted at . 
step 34 through the network 5 and is again received at step 35 at location A the 
30 received time stripped out (this being the time at which transmission first 
occurred) and the received time is compared again at step 37 with the local clock 
30 to provide, at step 38, a usable measure of latency of the network 5. 
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Whilst It woiitd be possible to transmit latency infomiation across tlie 
network so that each end used the same latency, in the case of network 
variations a similar latency measurement may be carried out from location B as 
indicated using the respective local clock 40 to derive a latency measure by way 
5 of steps 41 to 47 corresponding to those of steps 31 to 37. 

It will be noted that because only one clock is involved In determining the 
latency measurernent synchronisation of the clocks across the two 
communicating systems is not required. It will of course be appreciated that the 
packetisation need not necessarily be of specific clock time but may simply be a 
0 serial number which is transmitted and received and a look up table is used to 
determine the time of transmission of the series number packet for comparison 
. with the current time. 

Note that each end may perfomn a respective latency measurement In 
case there should be a difference between the latency being experienced across 
the network due to path variations in forward and reverse transmission paths. 
- . - -Once the latency in each direction has been determined, various methods 
of countering the latency problem may be used. Some examples of such methods 
are disclosed in our co-pending patent applications nos. EP02254458.9 and 
concurrently filed application No GB (our ref A30267). 

Considering Figure 5, in a typical haptic coupling across the network, 
local positions derived from the haptic output device sensors as Indicated at step 
51 and the remote position received from the network, step 52, are used to 
calculate differences, and -to provide difference vector (step 53) in respect of the 
X, y and z co-ordinates of the two haptic output devices. The coupling strength 
or resilience of the coupling between the two devices is then used (step 54) to 
calculate the force required to coerce the local haptic device to the relative 
position of the remote devjce (step 55) so that x, y and z vectors can be 
transmitted to provide the local force for motors at step 56. 

In the present invention, an alternative method of compensating for 
network latency is proposed which may be used instead of or as an enhancement 
of the latency compensation methods previously proposed. 

Thus rather than using only the received position of the remote 
handpiece, a prediction of where the next received position will be is used. There 
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are a number of methods proposed for predicting the position including 
predictions based on dynamic extrapolation from the current position and velocity, 
inriproving interpolation by measuring and transmitting contact forces from force 
sensors on the handpiece, building a model of the remote environment and force 
5 field modelling. 

Turning then to Rgure 6, in the first of the methods mentioned above, 
when the remote position is received from the network (step 60) It is stored in a 
position history record (61) which may simply be a rolling log of the last ^x" 
positions of the handpieces. The next position may now be predicted based upon 
0 the received position and a previous position using a known previous position or 
positions to determine the velocity (62) (estimated from previous motion) and the 
time interval in which the change occurred. Higher order terms may be taken into 
account for example. using acceleration (63), rate of change of acceleration and so 
on. This estimation works well with smooth movenrients where, the user is moving 
5 the handpiece in free space or where a surface is being traced for example but 
abrupt motions will be wrongly predicted such as the effect of a user hitting a 
hard surface at speed or a handpiece being used to feel the boundary of a solid 
object such as a cube, sphere or wall. 

Having determined the previous position, velocity and acceleration the 
expected change in position to the next received data packet can be calculated 
(73) with the input of the latency measure (64) as determined using the methods 
hereinbefore described, adapted by a factor (depending on the number of 
additional steps being taken between packets, and the predicted change in 
position may then be added to the current position received from the network 
(66). The local position determined from sensors in the local handpiece is now 
used to calculate the difference vector between the predicted position of the 
remote handpiece and the current position of the local handpiece (68). 

Other inputs such as the coupling strength setting between the two 
handpieces (69) may now be used in association with the difference vector to 
calculate the force required to coerce the local handpiece to the same relative 
position of the remote handpiece (70) and x, y and z signals are output to provide 
the local force to the motors of the local handpiece. 
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Note that the same algorithm Is implemented at both locations "A" and 
"B". The position transmitted from each end is the actual position at which the 
handpiece cun-ently resides and not the predicted position used for the calculation 
of local force to motors. Any error In the position prediction is of course not 
5 con-eetable in real time but adaptation of the forces to correct the effects of 
earlier prediction inaccuracies and move the local handpiece towards the remote 
handpiece position can be made. 

If the handpieces are equipped with force sensors then acceleration can 
be calculated directly rather that effecting a calculation from the positional data. 
10 The acceleration data is therefore available earlier if position and force data are 
transmitted between the remote and local environments. 

Thus referring to Figure 7, corresponding steps In the calculation have 
been described with reference to figure 6 and are not further elaborated here. 
However, here, additional steps are taken to modify the output to the motors of 

15 the haptic output device by introducing an element related to the force field 
model. Thus once the acceleration has been calculated at step 63 It is used 
together with the actual output force to derive a force field model. Thus at step 
74 the inertial component of the acceleration is subtracted from the force and is 
used to update a stored force field model with the force at the predicted position 

20 <75) . When the change in position has been determined, the force at the new 
position is looked up in the force field model and is added in at step 77 to the 
calculated force required to coerce the local handpiece towards the calculated 
position of the remote handpiece so that the accuracy of the force output to the 
X, y and z motors of the output device Is adjusted to take into account that force. 

25 in a further development in the present case, the local PC's each create a 

model of the space in which the effectors are moving and use the models to 
influence the local forces output to the motors. The space model data may be 
derived over time from a determination of positional and/or force data transmitted 
between the two haptic output devices or may be derived by sampling. A limrted 

30 data model may be constructed, particularly if sampling is used, say. storing data 
defining impedance or force presence at every tenth moveable point rather than at 
every point in the space model or . interpolating between positions with known 
values within the model. If a full computer model of the remote environment is 



wo 2005/041009 



PCT/GB2004/004025 



10 



available then remote interactions with the environment other than the effect of a 
remote user can be calculated. For example if the predicted position of the 
handpiece intercepts a position at which it Is known that the remote environment 
has a solid object of known mechanical properties then a reaction force can be 
5 predicted and added to the force applied to the local user via the local handpiece. 

So turning to Rgure 8, in which again steps corresponding to those 
described with respect to Figures 6 and 7 are correspondingly numbered and not 
further described, the inertial force from step 74 is compared with a threshold 
Which determines the presence or absence of an object at the position (78) and 
10 this determination is used to update the model of the operating space (79). Once 
the change in position has been calculated the predicted position is used to check 
for the presence of an object (80) and, if so, the reaction, fpr^e from the object is 
calculated (81) and then added in at step 77 as before. 

It will of course be noted that combinations of force modelling and space 
1 5 modelling may be used to influence the final output to the motors at step 71 

It is of course preferable if a full computer model of the remote space Is available. 

However, it is unlikely that the remote environment can be modelled with 
total accuracy, nor can it be guaranteed that objects within the remote 
environment have not been moved as a result of interactions between the users 
20 and the environment. The model can be updated over time from positional and 
force data, for example if past position data shows that at a certain position in a 
particular area a former movement of the handpiece resulted in the handpiece 
bouncing off then a tentative record of an object at that position can.be added to 
the model. Thus when the handpiece next moves to that location, a collision with 
25 the object in the model is simulated and a reaction force is added in to the output 
to the user even while the actual data reporting the collision is still in transit 
through the network. 

in further considerations, the models can be updated by averaging In 
Changes over time rather than by replacing current data model simulations 
30 completely. Accordingly, the model at the current position blends in to the old 
model and the more time that is spent at a position the more the new version 
corresponds with the remote environment and the less the old versions features in - 
the average. Although this can reduce the speed at which the model updates to 
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real changes, for example by an object being unpredictable In Its movement due 
to an interaction with the handpiece and other objects In the model, it increases 
the resistance of the model to the effects of noise, transients and other spurious 
effects of the connectionless transmissions. For linear force fields and other 
5 lineariy combinable effects it is possible to apply a decay factor 6 and on each 
time step multiply the existing value of the model at that position by 5-1 adding It 
in to 5 times the new version. 

Since it i8 .unlil(ely that a complete model of the remote environment can 
be determined and It would be impractical with current systems to sample the 

10 whole space in which a handpiece is working, the result, except in contrived 
circumstances, is likely to be a sparsely populated model of the environment. A 
partial model can be built up as the handpiece is moved vyithin the environment. 
Accordingly, interpolation between known values in the model is necessary. In 
one example this may be done by finding nearest know value points. 

1 5 Alternatively, a finite element model having less points than those present in the 
haptic operating space can be used. In this case the mode! has ceils much bigger 
than the minimal position disciimination of the haptic i/o- device but a blending 
technique between positions might be used to avoid a pixelated feeling to the 
forces felt by the user. In a more sophisticated arrangement the system may be 

20 arranged to determine the most likely arrangement of solid objects • within the 
operating space which match the values of the subset of points having known 
values. 

Other data reduction functions may be incorporated into the modelling - 
for example where textures are simulated they may be represented as periodic or 

25 stochastic functions having relatively few parameters, for example ridges can be 
specified by period, amplitude and ratio between ridge, slope and trough width. 
Thus vibrations from surface texture for example when tracing across a remote 
surface although difficult to predict by interpolation since their small scale means 
that a number of bumps of a finely textured surface can be moved over during the 

30 network latency delay, may be simulated from bulk texture parameters held within 
the data model. 

Other methods of networking latency measurement (e.g. ISDN, TCP over 
IP or RS232 serial over a modem to modem link over PSTN) could be used instead 
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of UDP. Other methods of network latency measurert^ent (e.g. 'ping' time, 
network performance metrics from other computers on the network, or single 
direction measurement by synchronised clocks) could be used. 
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